{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fa1a24c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install arize-phoenix openai tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d58c5245e6d7811f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import phoenix as px\n",
    "from phoenix.experimental.evals.functions import llm_classify\n",
    "from phoenix.experimental.evals.models import OpenAIModel\n",
    "from phoenix.experimental.evals.templates.default_templates import (\n",
    "    HALLUCINATION_PROMPT_RAILS_MAP,\n",
    "    HALLUCINATION_PROMPT_TEMPLATE,\n",
    "    QA_PROMPT_RAILS_MAP,\n",
    "    QA_PROMPT_TEMPLATE,\n",
    "    RAG_RELEVANCY_PROMPT_RAILS_MAP,\n",
    "    RAG_RELEVANCY_PROMPT_TEMPLATE,\n",
    ")\n",
    "from phoenix.session.evaluation import (\n",
    "    get_qa_with_reference,\n",
    "    get_retrieved_documents,\n",
    "    log_evaluations,\n",
    ")\n",
    "from phoenix.trace.exporter import HttpExporter\n",
    "from phoenix.trace.span_evaluations import DocumentEvaluations, SpanEvaluations\n",
    "from sklearn.metrics import ndcg_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb6dec6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a47cb56dc6a1d7b9",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Key Takeaways\n",
    "\n",
    "1. Evaluations can be performed on individual documents, not just spans. In general, as long as the subject of an evaluation is an identifiable entity, we can update Phoenix to ingest it. So having the appropriate identifiers is important for the evaluation results.\n",
    "2. Each evaluation result consists of a combination of values under the reserved keys: `label` (str), `score` (float), and `explanation` (str).\n",
    "3. Each set of evaluations should have a name, which is an arbitrary string. Evaluations with the same name will be grouped together when computing aggregate metrics. The name is also used to identify different evaluations for the same subject when they are displayed in the UI.\n",
    "4. We need `score` to compute averages, but may need a positive class value to convert `label` to `score`, e.g. setting `score` = 1 if `label` == \"relevant\". This is done manually here, but may be automated in the future.\n",
    "5. Document retrieval metrics such as NDCG can be computed by Phoenix automatically but its UI is not ready yet, so we compute them manually here and show how arbitrary evaluation results can be ingested into Phoenix, by considering them as annotations.\n",
    "6. Evaluation results are ingested via the HttpExporter. A helper function is provided for doing that.\n",
    "7. Inputs to the evaluation templates are extracted from the active Phoenix session via helper functions. These helper functions also put in place the appropriate identifiers for ingesting the evaluation results back to Phoenix.\n",
    "8. A more complicated evaluation shown here is the `Q&A Correctness`, which requires merging details from separate spans, i.e. the retrieved documents from the retriever span, and LLM response from a different span. This is done manually here, but we may automate it in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b43166d02c26e8d",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Launch Phoenix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "549daf5477419ea5",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We start Phoenix with a small dataset of traces to represent the scenario where traces have already been exported into Phoenix (e.g. from tracing the user's applications), and after seeing the spans in Phoenix, the user wants to perform evaluations on them. These example spans in particular comes from an RAG application, so some spans contain a list of retrieved documents. Our user is interested in how relevant each document is relative to the question that was asked, how to compute aggregate metrics on these retrievals, and how the document level evaluations impact the overall qualities of the Q&A spans."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "420409a6557bf2c7",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "One key concept that's demonstrated in the notebook is that the subject of an evaluation is not limited to only spans. Subcomponents of a span such individual retrieved documents can be evaluated as well. As long as appropriate identifiers are present for joining the evaluations back to their subjects, the evaluations can be ingested, and Phoenix can perform further analysis on them such as aggregation and metrics. This allows the user to decompose the evaluation of a span into its subcomponents, perform evaluations on them separately, and then aggregate the results back to the span level. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial_id",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = px.load_example_traces(\"llama_index_rag\")\n",
    "px.launch_app(trace=ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6119e74cc6270234",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Once Phoenix is running in the background, we can apply helper functions to extract relevant details needed to perform various kinds of evaluations. The details extracted are dependent on the input requirements of each type of evaluation, and the helper functions will put in place the appropriate identifiers for joins later on when the eval results are ingested back to Phoenix."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e159ff1afd12933",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Ingestion will take place via HTTP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7479b85bbed6120",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We may streamline this process in the future, but for the time being we'll just manually re-use the existing exporter that we have for spans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1dc6136806c8435",
   "metadata": {},
   "outputs": [],
   "source": [
    "exporter = HttpExporter()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1362576ff0fe4e2c",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Extract Retrieved Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed50f807f2d8e690",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "First we will evaluate the relevance of the documents that were retrieved by the RAG model. The helper function `get_retrieved_documents` will extract the relevant details from the spans and the retrieved documents, and put them in a dataframe that can be used for evaluation.  Note that the input to the helper function is the current session object, but it's possible that the session was started in a different notebook and the user doesn't have access to it here, so we'll need to provide a solution in a future update if that becomes a real use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2c45b85c6644735",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_documents = get_retrieved_documents(px.active_session())\n",
    "retrieved_documents"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3af312c6ebfc0dc4",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "The result dataframe is indexed by the span ID and document position. The index is important because `llm_classify` will return a result dataframe preserving the index of the input dataframe, and during ingestion we'll use the index values of each row as the identifier of each evaluation result. `input` is the question that was asked, `reference` is the retrieved document. These columns are named as such because those are the template variable names. The `document_score` column is the score that the RAG model assigned to the document. The score is useful for sorting the documents when computing a metric such as NDCG. We also added trace_id here because later on we'll show an eval that requires us to join the documents to a different span containing the final answer from the LLM."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ac938a5c199dc82",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Set Up OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e14465175520ce42",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "import openai\n",
    "\n",
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "openai.api_key = openai_api_key\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de9664171d3e33b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = OpenAIModel(model_name=\"gpt-3.5-turbo\")\n",
    "model(\"hi\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d694213dcf35676f",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Evaluate Document Relevance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b79f8d7faccb740",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We'll use the `llm_classify` function as usual, but add a new column named `score` containing binary integers, i.e. 1 or 0. The importance of the `score` column is to let us \"numerify\" the evaluation results, so we can do aggregations such as averaging. This is actually not a simple task in the general case, because it requires knowing which label is the positive one, i.e. the one receiving the value 1 instead of 0. For a general eval, the user would have to tell us how to compute the score, but in this case we know that the label `relevant` should have the score of 1."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d335ca5cd0f6635",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "`label`, `score`, and `explanation` are the principal keywords for the values of an evaluation result. Every evaluation result to be ingested into Phoenix must have one of these values filled out. On the other hand, it's possible to have missing values for example when `llm_classify` is terminated by user before completion. In that case this notebook will not crash and those results with all missing values won't be ingested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "516dc273735ad00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_documents_eval = llm_classify(\n",
    "    retrieved_documents,\n",
    "    model,\n",
    "    RAG_RELEVANCY_PROMPT_TEMPLATE,\n",
    "    list(RAG_RELEVANCY_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,\n",
    ")\n",
    "retrieved_documents_eval[\"score\"] = (\n",
    "    retrieved_documents_eval.label[~retrieved_documents_eval.label.isna()] == \"relevant\"\n",
    ").astype(int)\n",
    "retrieved_documents_eval.to_parquet(\n",
    "    \"llama_index_rag_with_rerank.documents_eval.parquet\"\n",
    ")  # not required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54be94acd3b123cd",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We're saving the results in parquet solely as a convenience for the users of this notebook. This is not a required step in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f341795ae24ca024",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_documents_eval = pd.read_parquet(\n",
    "    \"llama_index_rag_with_rerank.documents_eval.parquet\"\n",
    ")  # not required\n",
    "retrieved_documents_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "357fe94b02b22a6b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Merge Evaluation Results Back to the Original Data to Compute Retrieval Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6307218ea5f50e6e",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Retrieval metrics such as NDCG and Precision@k can be computed by Phoenix automatically, but the UI is not ready yet, so for demonstration purposes, we'll carry out the calculations by hand, and ingest the result as separate evaluations. This also doubles as a demonstration of how to ingest evaluations that are not generated by `llm_classify`, by considering evaluation results as general annotations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fbe1c24d6e46768",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "First we'll merge the retrieved documents with the evaluation results. This is needed to compute NDCG based on the original document scores assigned by the retriever. The `eval_` prefix is added to the columns from the evaluation results just for clarity, but is not required."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3bd04b678c9d18c",
   "metadata": {},
   "outputs": [],
   "source": [
    "combined = pd.concat([retrieved_documents, retrieved_documents_eval.add_prefix(\"eval_\")], axis=1)\n",
    "combined"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b162eccd6c69aa7f",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Compute NDCG@2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b7a7c22e0634050",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This one has a small wrinkle for handling missing values in case `llm_classify` was terminated before completion. Otherwise, it's a pretty straightforward application of `sklearn.metrics.ndcg_score`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77d9fdebd46d268b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _compute_ndcg(df: pd.DataFrame, k: int):\n",
    "    \"\"\"Compute NDCG@k in the presence of missing values\"\"\"\n",
    "    n = max(2, len(df))\n",
    "    eval_scores = np.zeros(n)\n",
    "    doc_scores = np.zeros(n)\n",
    "    eval_scores[: len(df)] = df.eval_score\n",
    "    doc_scores[: len(df)] = df.document_score\n",
    "    try:\n",
    "        return ndcg_score([eval_scores], [doc_scores], k=k)\n",
    "    except ValueError:\n",
    "        return np.nan\n",
    "\n",
    "\n",
    "ndcg_at_2 = pd.DataFrame({\"score\": combined.groupby(\"context.span_id\").apply(_compute_ndcg, k=2)})\n",
    "ndcg_at_2.to_parquet(\"llama_index_rag_with_rerank.ndcg_at_2.parquet\")  # not required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc063f28a243f37e",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We're saving the results in parquet solely as a convenience for the users of this notebook. This is not a required step in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8032851d13b63d55",
   "metadata": {},
   "outputs": [],
   "source": [
    "ndcg_at_2 = pd.read_parquet(\"llama_index_rag_with_rerank.ndcg_at_2.parquet\")  # not required\n",
    "ndcg_at_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8d5816954fbaa4d",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Compute Precision@3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f397e5d79eebf00",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Precision@k is really simple. We change the value for `k` here just for fun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3167f4675c7313",
   "metadata": {},
   "outputs": [],
   "source": [
    "precision_at_3 = pd.DataFrame(\n",
    "    {\n",
    "        \"score\": combined.groupby(\"context.span_id\").apply(\n",
    "            lambda x: x.eval_score[:3].sum(skipna=False) / 3\n",
    "        )\n",
    "    }\n",
    ")\n",
    "precision_at_3.to_parquet(\"llama_index_rag_with_rerank.precision_at_3.parquet\")  # not required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "584108d93416a0e1",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We're saving the results in parquet solely as a convenience for the users of this notebook. This is not a required step in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c1d31d1d1c95429",
   "metadata": {},
   "outputs": [],
   "source": [
    "precision_at_3 = pd.read_parquet(\n",
    "    \"llama_index_rag_with_rerank.precision_at_3.parquet\"\n",
    ")  # not required\n",
    "precision_at_3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1819b377e7602361",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Merge Documents from Retrieval Spans to Q&A Spans (to Compute Q&A Correctness)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c65f739233949b0",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Trace ID is what we use to merge together details from separate spans, i.e. the retrieved documents from the retriever span, and LLM response from a different span. This is done manually here, but we may automate it in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb27fd4724e0e27e",
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_df = get_qa_with_reference(px.active_session())\n",
    "qa_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4084449c986aed8",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Evaluate Q&A Correctness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae507af54ce886a",
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_correctness_eval = llm_classify(\n",
    "    qa_df,\n",
    "    model,\n",
    "    QA_PROMPT_TEMPLATE,\n",
    "    list(QA_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,\n",
    ")\n",
    "qa_correctness_eval[\"score\"] = (\n",
    "    qa_correctness_eval.label[~qa_correctness_eval.label.isna()] == \"correct\"\n",
    ").astype(int)\n",
    "qa_correctness_eval.to_parquet(\n",
    "    \"llama_index_rag_with_rerank.qa_correctness_eval.parquet\"\n",
    ")  # not required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1bd6821c887a0",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We're saving the results in parquet solely as a convenience for the users of this notebook. This is not a required step in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2041d2dcc7d02322",
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_correctness_eval = pd.read_parquet(\n",
    "    \"llama_index_rag_with_rerank.qa_correctness_eval.parquet\"\n",
    ")  # not required\n",
    "qa_correctness_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a88f90ea9c24832b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Evaluate Hallucination"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4d1b39db998ada0",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Note that we set `factual` to be the positive class, so the score will be 1 if the label is `factual`, so that a higher average score indicates a more positive outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "946b5aad5d72c1f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "hallucination_eval = llm_classify(\n",
    "    qa_df,\n",
    "    model,\n",
    "    HALLUCINATION_PROMPT_TEMPLATE,\n",
    "    list(HALLUCINATION_PROMPT_RAILS_MAP.values()),\n",
    "    provide_explanation=True,\n",
    ")\n",
    "hallucination_eval[\"score\"] = (\n",
    "    hallucination_eval.label[~hallucination_eval.label.isna()] == \"factual\"\n",
    ").astype(int)\n",
    "hallucination_eval.to_parquet(\n",
    "    \"llama_index_rag_with_rerank.hallucination_eval.parquet\"\n",
    ")  # not required"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20625b9c551db24b",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "We're saving the results in parquet solely as a convenience for the users of this notebook. This is not a required step in general."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8b1a6d7143c986e",
   "metadata": {},
   "outputs": [],
   "source": [
    "hallucination_eval = pd.read_parquet(\n",
    "    \"llama_index_rag_with_rerank.hallucination_eval.parquet\"\n",
    ")  # not required\n",
    "hallucination_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baea4ca8",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluations = [\n",
    "    DocumentEvaluations(eval_name=\"Relevance\", dataframe=retrieved_documents_eval),\n",
    "    SpanEvaluations(eval_name=\"Hallucination\", dataframe=hallucination_eval),\n",
    "    SpanEvaluations(eval_name=\"Q&A Correctness\", dataframe=qa_correctness_eval),\n",
    "    SpanEvaluations(eval_name=\"ndcg@2\", dataframe=ndcg_at_2),\n",
    "    SpanEvaluations(eval_name=\"precision@3\", dataframe=precision_at_3),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28e44983",
   "metadata": {},
   "outputs": [],
   "source": [
    "log_evaluations(*evaluations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f94f8f9",
   "metadata": {},
   "source": [
    "# Colocate evaluation results with the traces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44209557",
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.trace import TraceDataset\n",
    "\n",
    "trace_dataframe = px.Client().get_spans_dataframe()\n",
    "trace_ds = TraceDataset(\n",
    "    trace_dataframe,\n",
    "    evaluations=evaluations,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e6cd07b",
   "metadata": {},
   "outputs": [],
   "source": [
    "trace_ds.get_evals_dataframe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "370ca233",
   "metadata": {},
   "outputs": [],
   "source": [
    "trace_ds.get_spans_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83dd4cd21c966504",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# End Session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fdea46ec99abef2",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "This is commented out but shows how to terminate Phoenix running in the background."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5842da4238a93554",
   "metadata": {},
   "outputs": [],
   "source": [
    "# px.active_session().end()"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
